Many of today's computers are accessible through graphical user interfaces (GUIs). The use of GUIs has greatly increased the usability of computers for the general population and has greatly enhanced the role computers play in everyday life. Over time, the application software available to computer users has become increasingly complex, with an increasing number of capabilities, options, and resources presented to users of each class of software application, of which the class of word-processing software applications may be considered as an example. A problem that tends to accompany any increase in capabilities, options, or resources in a software application is that it generally becomes more difficult for the user to find and operate the new capabilities, options, or resources.
Software designers have responded to such problems and challenges generally by adding help and search capabilities to the software. These allow the user to search for specific software features or resources that they wish to utilize. There are several deficiencies in the conventional help and search based mechanisms. Current mechanisms often rely on written communication of ideas between the user and the software, and this presupposes that the user is able to read and write (and type) a language used by the software. These mechanisms generally require the user to understand the nature of search and queries as implemented by software components, often in terms of Boolean logic. Users are generally required to understand specific terminology and concepts used by the designers of the software application. These mechanisms also generally require the user to switch tasks (e.g. between working on a task and searching for how to accomplish a subtask and learning about a software application), and often require the user to switch between a pointer (e.g. mouse) based task such as the assembly of a figure and a keyboard based task such as searching for components related to ones they have already assembled.
In addition to increasing application complexity, the need for improved searching techniques has grown in recent years due to the much increased volume of information available to computer applications and computer users. The availability of information has grown most significantly because of the increased sharing of information amongst computers and computer users.
In recent years, there has been a tremendous proliferation of computers connected to a global network known as the Internet. A “client” computer connected to the Internet can download digital information from “server” computers connected to the Internet. Client application software executing on client computers typically accepts commands from a user and obtains data and services by sending requests to server applications running on server computers connected to the Internet. A number of protocols are used to exchange commands and data between computers connected to the Internet. The protocols include the File Transfer Protocol (FTP), and the Hypertext Transfer Protocol (HTTP).
The HTTP protocol is used to access data on the World Wide Web, often referred to as “the Web.” The World Wide Web is an information service on the Internet providing documents and links between documents. The World Wide Web is made up of numerous Web sites around the world that maintain and distribute electronic documents. A Web site may use one or more Web server computers that store and distribute documents in one of a number of formats including the Hypertext Markup Language (HTML). An HTML document contains text and metadata or commands providing formatting information. HTML documents also include embedded “links” that reference other data or documents located on any Web server computers. The referenced documents may represent text, graphics, audio, or video in respective formats.
A Web browser is a client application or operating system utility that communicates with server computers via FTP, HTTP, and/or other protocols. Web browsers receive electronic documents from the network and present those documents to a user. FIREFOX™, which is available from the Mozilla Foundation and INTERNET EXPLORER®, which is available from Microsoft Corporation, of Redmond, Wash., are examples of popular Web browser applications.
In addition to data and metadata, HTML documents can contain embedded software components containing program code that perform a wide variety of operations. These software components expand the interactive ability of an HTML document's user interface. The components can perform other operations, such as manipulating data and playing audio or video clips. ActiveX is a specification developed by Microsoft Corporation for creating software components that can be embedded into an HTML document. JAVA™ is a well-known programming language that can be used to develop components called “applets,” which are transmitted with HTML documents from Web servers to client computers. JAVASCRIPT™ and VBScript are scripting languages that are also used to extend the capabilities of HTML. JAVASCRIPT and VBScript scripts are embedded in HTML documents. A browser executes each script as it reaches the position in the script during interpretation of the HTML document. Scripts loaded during interpretation of the document may modify the document if the browser supports dynamic HTML (DHTML). Scripts may respond to user activity (pointer events or keyboard events), may post data to the server, and may request and receive data from the server. They may schedule additional scripts to execute upon the receipt of certain requested data from the server, allowing for asynchronous, data-driven events to occur.
The use of query graphs to facilitate search, and, in particular, the presentation of such query graphs in a user interface so as to allow the user to navigate a graph of potential search queries has been proposed as a mechanism to partly alleviate some difficulties users may face in constructing appropriate query strings. Query graphs do not, however, alleviate the need for users to understand the query mechanism or details regarding the search component. A query graph may require a user to become more familiar with the inner structure of the search component, while at the same time providing the user with a richer search capability in certain usage scenarios. In addition, a query graph does not alleviate others of the search-related difficulties cited above, including those of language barriers and task switching.
Many, if not most, search applications in the current art are text (language) based; they are driven by and respond to textual queries. This is true both for active search applications in which the user enters specific search queries (e.g., the GOOGLE™ search application, from Google, Inc., of Sunnyvale Calif.), for various categorized electronic data sources (e.g., the YAHOO!™ Web directory, and the WIKIPEDIA® on-line encyclopedia from the Wikimedia Foundation) and for passive search applications which monitor user activities and report relevant search results (e.g., Watson contextual search software, from Media River, Inc.). At the same time, a growing amount of computer-user interaction and content retrieved by users through computer interaction is non-text-based and/or non-language-based. This is true in GUIs for desktop components and sub-components, as illustrated by the increased use of icons to represent resources, applications, and actions within applications. It is also true for content retrieved, stored, and accessed through computers, with increased user interest in content in the form of music, images, or video.
There is therefore a need, not met by the current art, for new methods and mechanisms for search that are not language based.